Instance Smoothed Contrastive Learning for Unsupervised Sentence Embedding

Contrastive learning-based methods, such as unsup-SimCSE, have achieved state-of-the-art (SOTA) performances in learning unsupervised sentence embeddings. However, previous studies, each embedding used for contrastive only derived from one instance, and we call these embeddings instance-level In other words, is regarded a unique class of its own, which may hurt the generalization performance. this study, propose IS-CSE (instance smoothing embedding) to smooth boundaries feature space. Specifically, retrieve dynamic memory buffer according semantic similarity get positive group. Then group are aggregated by self-attention operation produce smoothed instance further analysis. We evaluate our method on standard text (STS) tasks achieve an average 78.30%, 79.47%, 77.73%, 79.42% Spearman’s correlation base BERT-base, BERT-large, RoBERTa-base, RoBERTa-large respectively, 2.05%, 1.06%, 1.16% 0.52% improvement compared unsup-SimCSE.